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Abstract 


This  system  manual  provides  an  overview  of  software  developed  to  support  the 
empirical  investigation  of  a  simulated  user  interface  for  an  Advanced  Integrated  Multi¬ 
sensor  Surveillance  (AIMS)  system  (formerly  known  as  the  Enhanced  Low-Light 
Level  Visible  and  Infrared  Surveillance  System  -  ELVISS).  The  AIMS  system  is  an 
electro-optical  imaging  system  being  developed  by  the  Defence  Research  and 
Development  Canada  (DRDC)  -  Valcartier  to  enhance  the  capability  of  search  and 
rescue  (SAR)  crews  to  operate  effectively  at  night  and  in  degraded  weather  conditions. 
In  order  to  ensure  that  a  SAR  operator  would  be  able  to  use  the  system  effectively  and 
with  a  minimal  amount  of  training,  a  prototype  human-machine  interface  (HMI)  was 
developed  to  evaluate  design  concepts.  The  latest  development  phase  added  important 
tracking  and  motion-related  functionality  (amongst  other  things)  to  the  system  and 
gave  it  a  new  name  AIMSsim. 


Resume 


Le  manuel  de  Tutilisateur  fournit  une  vue  d’ensemble  sur  Tutilisation  du  logiciel 
developpe  pour  appuyer  la  recherche  empirique  d’une  interface-utilisateur  de 
simulation  pour  le  systeme  AIMS  -  systeme  multicapteur  integre  de  pointe  pour  la 
surveillance  (anciennement  connu  sous  Tappellation  ELVISS  -  systeme  perfectionne 
de  surveillance  a  intensification  de  lumiere  visible  et  a  infrarouge).  Le  systeme  AIMS 
est  un  systeme  d’imagerie  electro-optique  mis  au  point  par  Recherche  et 
Developpement  pour  la  defense  Canada  (RDDC)  -  Valcartier  pour  ameliorer  les 
capacites  de  Tequipe  de  recherche  et  sauvetage  (SAR).  Elle  pourra  done  effectuer  ses 
missions  de  fafon  plus  efficace  dans  l’obscurite  et  dans  de  mauvaises  conditions 
meteorologiques.  Afm  de  s’assurer  que  Toperateur  SAR  est  capable  d’utiliser 
adequatement  le  systeme  et  ce  avec  une  formation  minimale,  un  prototype  d’interface 
homme-machine  (HMI)  a  ete  elabore  pour  evaluer  les  principes  de  conception.  La 
demiere  phase  d’elaboration  a,  entres  autres,  permis  de  munir  le  systeme  d’une 
importante  fonction  de  localisation  et  d’une  fonction  relative  au  mouvement.  Ces 
ajouts  lui  ont  valu  une  nouvelle  appellation,  AIMSsim. 
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Executive  summary 


Introduction 

This  document  provides  an  overview  of  software  developed  to  support  the  empirical 
investigation  of  a  simulated  user  interface  for  an  Advanced  Integrated  Multi-sensor 
Surveillance  (AIMS)  system  (formerly  known  as  the  Enhanced  Low-Light  Level 
Visible  and  Infrared  Surveillance  System  -  ELVISS). 

A  multi-sensor  surveillance  system,  the  Advanced  Integrated  Multi-sensor 
Surveillance  (AIMS)  system,  is  being  developed  to  increase  the  capability  of  Search 
and  Rescue  (SAR)  and  Maritime  patrol.  The  AIMS  system  will  enhance  the  capability 
of  SAR  particularly  at  night  and  in  poor  weather.  Earlier  versions  of  AIMS  were  the 
Airborne  Laser  Based  Enhanced  Detection  and  Observation  System  (ALBEDOS),  and 
the  Enhanced  Low-Light  Level  Visible  and  InfraRed  Surveillance  System  (ELVISS). 
The  AIMS  system  advanced  through  the  integration  of  four  sensors  into  a  single 
gimball.  A  research  platform  that  simulates  use  of  the  airborne  sensor  interface  and 
controls  has  been  developed  at  Defence  Research  and  Development  Canada  (DRDC) 
to  support  evaluation  of  interface  design  concepts  and  to  address  human  performance 
issues  related  to  operating  the  AIMS  and  similar  electro-optical  imaging  systems. 

In  order  to  investigate  human  performance  ensure  that  a  SAR  operator  would  be  able 
to  use  the  system  effectively  and  with  a  minimal  amount  of  training,  a  prototype 
human-machine  interface  (HMI)  was  developed  to  evaluate  design  concepts.  The 
VAPS  HMI  prototype  (ELVISS),  developed  for  the  Silicon  Graphics,  Inc.’s  (SGI) 
platform,  provided  a  cost  effective  method  for  evaluating  the  impact  of  design 
characteristics  of  dual  sensor  systems  on  operator  performance.  However  the 
capability  was  limited  and  the  architecture  was  not  designed  to  support  systematic 
investigation  of  the  usefulness  of  the  proposed  system  under  different  conditions  or  to 
manipulate  the  sensor  and  interface  characteristics. 


Results 

The  ELVISS  VAPS  prototype  was  therefore  extensively  enhanced  to  allow  the 
empirical  investigation  of  different  interface  and  sensor  characteristics  on  search  and 
detection  capability  under  different  environmental  conditions.  Included  in  this  upgrade 
was  a  Scenario  Generation  Environment  (SGE)  that  provided  user-friendly  capability 
for  generating  scenarios.  Nonetheless,  despite  the  increased  versatility  of  the  prototype 
the  requirements  of  a  specific  experimental  design  required  that  LUA  scripting 
(www.lua.org)  be  used  to  make  additional  modifications  to  the  software.  While  further 
development  drastically  expanded  the  LUA  scripting  capabilities,  the  SGE  was  not 
similarly  extended  and  some  of  its  scenario-generation  capabilities  became 
incompatible  with  the  prototype.  Thus  LUA  scripting  in  now  the  primary  mode  of 
control  of  the  prototype. 

The  prototype  HMI  was  then  ported  to  run  on  Microsoft  Windows  XP,  and  required 
replacing  the  VAPS  and  SGI  Performer™  with  equivalent  functionality  using  the 
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OpenSceneGraph  open  source  graphics  library.  The  robustness  and  traceability  of  the 
system  were  also  significantly  improved.  The  latest  development  phase  added 
important  tracking  and  motion-related  functionality  (amongst  other  things)  to  this  new 
system  and  gave  it  a  new  name  AIMSsim. 

Significance 

The  experimental  research  platform  at  DRDC  provides  a  means  for  ensuring  that  the 
user  is  an  integral  part  of  the  design  process  and  optimal  design  from  the  user’s 
perspective  is  obtained.  As  technology  advances  and  systems,  like  the  AIMS,  become 
more  complex  for  an  operator  to  use,  user-machine  system  design  becomes  more 
critical  and  challenging.  The  continued  development  and  upgrade  of  the  AIMSsim 
research  platform  provides  the  experimenter  with  an  appropriate  level  of  simulation 
detail  to  conduct  human  peformance  analyses  which  in  turn  delivers  up-to-date 
knowledge  and  advice  on  the  design  of  sensor  surveillance  systems  to  the  military 
stakeholder.  A  User  Manual  (Schoenbaum,  2007a)  describing  functionality  and  how  to 
use  the  system,  and  a  Final  Report  (Schoenbaum,  2007b)  that  summarizes  the  work 
performed  and  makes  recommendation  for  future  work,  are  also  associated  with  this 
document. 
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Introduction 

Le  manuel  foumit  les  instructions  sur  Installation  et  l’utilisation  du  logiciel 
developpe  pour  appuyer  la  recherche  empirique  d’un  interface-utilisateur  de 
simulation  pour  le  systeme  AIMS  -  systeme  multicapteur  integre  de  pointe  pour  la 
surveillance  (anciennement  connu  sous  l’appellation  ELVISS  -  systeme  perfections 
de  surveillance  a  intensification  de  lumiere  visible  et  a  infrarouge). 

Systeme  multicapteur  de  surveillance,  le  AIMS  est  en  cours  de  developpement  pour 
ameliorer  les  capacites  de  recherche  et  sauvetage  (SAR)  et  de  la  patrouille  maritime. 

Le  systeme  AIMS  optimisera  les  capacites  de  SAR  plus  particulierement  la  nuit  et  dans 
de  mauvaises  conditions  meteorologiques.  D’autres  versions  du  systeme  AIMS  avaient 
deja  ete  developpees,  soit  le  systeme  laser  aeroporte  perfections  de  detection  et 
d’observation  (ALBEDOS)  et  le  systeme  perfections  de  surveillance  a  intensification 
de  lumiere  visible  et  a  infrarouge  (ELVISS).  Le  systeme  AIMS  est  superieur  a  ses 
predecesseurs  grace  a  L  integration  de  quatre  capteurs  dans  un  seul  cardan.  Une 
plateforme  de  recherche  qui  simule  l’utilisation  et  la  commande  de  interface  de 
capteur  aeroporte  a  ete  elaboree  par  Recherche  et  Developpement  pour  la  defense 
Canada  (RDDC)  afm  d’appuyer  L  evaluation  des  principes  de  conception  et  pour 
aborder  les  questions  relatives  au  rendement  humain  lie  a  l’utilisation  du  systeme 
AIMS  et  de  systemes  d’imagerie  electro-optique  semblables. 

Afm  de  s’assurer  que  l’operateur  SAR  est  capable  d’utiliser  adequatement  le  systeme, 
et  ce  avec  us  formation  minimale,  un  prototype  d’interface  homme-machine  (HMI)  a 
ete  elabore  pour  evaluer  les  principes  de  conception.  Le  prototype  VAPS  HMI 
(ELVISS),  developpe  pour  la  plateforme  de  Silicon  Graphics,  Inc.  (SGI),  a  foumi  us 
methode  economique  pour  evaluer  l’impact  sur  le  rendement  de  l’operateur  des 
caracteristiques  de  conception  des  systemes  de  capteurs  jumeles.  Le  systeme  a 
demontre  que  ses  capacites  etaient  limitees  et  que  son  architecture  n’avait  pas  ete 
conguc  pour  permettre  une  recherche  systematique  de  l’utilite  du  systeme  propose  dans 
differentes  conditions  ou  pour  manipuler  les  caracteristiques  des  capteurs  et  de 
1’ interface. 

RESULTATS 

Le  prototype  ELVISS  VAPS  a  done  ete  grandement  ameliore  pour  permettre  la 
recherche  experimentale  sur  des  interfaces  differentes  et  des  caracteristiques  de 
capteurs  pour  la  recherche  et  la  detection  dans  diverses  conditions  environnementales. 
Cette  version  amelioree  incluait  egalement  un  environnement  de  generation  de 
scenarios  (Scenario  Generation  Environment  [SGE])  qui  foumissait  une  capacite 
conviviale  pour  generer  des  scenarios.  Finalement,  malgre  la  polyvalence  amelioree  du 
prototype,  les  exigences  d’une  conception  experimentale  specifique  demandaient 
qu’un  script  LUA  (www.lua.org)  soit  utilise  pour  apporter  des  modifications 
supplementaires  au  logiciel.  Tandis  que  des  nouveaux  progres  elargissaient  les 
capacites  de  script  LUA,  le  SGE  n’evoluait  pas  de  la  meme  fa5on  et  quelques-unes  de 
ces  capacites  de  scenarisation  sont  meme  devenues  incompatibles  avec  le  prototype. 
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C’est  pourquoi  le  script  LUA  est  maintenant  le  principal  mode  de  controle  du 
prototype. 

Le  prototype  HMI  a  alors  ete  adapte  pour  fonctionner  avec  Microsoft  Windows  XP,  et 
a  demande  le  remplacement  de  VAPS  (Virtual  Applications  Builder)  et  de  SGI 
Performer™  par  une  fonctionnalite  equivalente  utilisant  la  graphitheque  de  source 
ouverte  OpenSceneGraph.  La  robustesse  et  la  tragabilite  du  systeme  ont  egalement  ete 
ameliorees  de  fag  on  significative.  Le  derniere  phase  de  developpement  a,  entres  autres, 
munie  le  systeme  d’une  importante  fonction  de  localisation  et  d’une  fonction  relative 
au  mouvement.  Ces  ajouts  lui  ont  valu  une  nouvelle  appellation,  AIMSsim. 

PORTEE 

La  plateforme  de  recherche  experimental  a  RDDC  a  fourni  des  moyens  pour  s’ assurer 
que  l’utilisateur  fait  partie  du  processus  de  conception  et  que  la  perspective  de  ce 
dernier  sur  la  conception  optimale  est  connue.  Tout  comme  la  technologie,  les 
systemes  comme  AIMS  se  perfectionnent  et  deviennent  de  plus  en  plus  complexes  a 
utiliser  pour  un  operateur;  la  conception  du  systeme  utilisateur-machine  devient  de 
plus  en  plus  important  et  impose  de  nouveaux  defis.  Le  developpement  continu  et  la 
mise  a  niveau  de  la  plateforme  de  recherche  AIMSsim  procurent  a  Texperimentateur 
assez  de  detail  sur  la  simulation  pour  effectuer  des  analyses  sur  le  rendement  humain 
qui,  a  leurs  tours,  foumissent  aux  intervenants  militaires  une  connaissance  actuelle  et 
des  recommandations  sur  la  conception  de  systemes  de  capteurs  de  surveillance.  Un 
manuel  de  systeme  (Schoenbaum,  2007a)  decrivant  Tarchitecture  et  les  capacites  du 
systeme  et  un  rapport  final  (Schoenbaum,  2007b)  resumant  les  travaux  effectues  et 
faisant  des  recommandations  pour  de  futures  recherches,  sont  egalement  lies  au 
present  document. 
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Foreword: 


This  version  of  the  System  Manual  incoiporates  all  changes  to  the  DRDC  AIMS  HMI 
Experimental  Prototype,  as  of  March  2006.  This  document  is  based  on  the  System 
Manual,  created  by  the  HFE  Group,  and  the  Experimenter’s  Guide,  provided  by 
DRDC.  The  HFE  Group  and  CMC  are  no  longer  responsible  for  the  content  of  this 
document.  Along  with  the  User’s  Manual,  this  manual  provides  a  good  introduction 
and  reference  to  the  DRDC  AIMS  HMI  Experimental  Prototype  software,  AIMSsim. 
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1  Introduction 


1.1  General 

Defence  Research  and  Development  Canada  (DRDC)  is  developing  a  fly  able 
prototype  of  an  Advanced  Integrated  Multi-sensor  Surveillance  (AIMS)  system 
(formerly  known  as  the  Enhanced  Low-Light-Level  Visible  and  Infrared  Surveillance 
System  -  ELVISS).  DRDC  has  contracted  out  the  development,  and  subsequent 
enhancement  of  the  capabilities,  of  the  AIMS  human-machine  interface  (HMI) 
prototype,  to  allow  the  empirical  investigation  of  different  interface  and  sensor 
characteristics  on  search  and  detection  capability  under  a  variety  of  simulated 
environmental  conditions. 


1.2  Background 

The  Department  of  National  Defence  (DND)  has  identified  a  requirement  to  enhance 
the  capabilities  of  Search  and  Rescue  (SAR)  operators  to  conduct  operations  at  night 
and  under  degraded  weather  conditions.  To  this  end.  Defence  Research  and 
Development  Canada  (DRDC)  is  developing  a  multi-sensor  system  composed  of  an 
Active  Gated  TV  (AGTV)  and  a  thermal  Infrared  (IR)  imaging  system.  By 
coordinating  the  use  of  a  pulsed  laser  illuminator  and  AGTV  camera,  the  AGTV 
component  of  AIMS  provides  effective  imaging  in  the  absence  of  ambient  light.  In 
addition,  the  active  range  gate  allows  the  AGTV  system  to  penetrate  meteorological 
phenomena  such  as  fog,  snow  and  rain  much  more  effectively  than  a  F LIR  camera. 

The  F LIR  camera  is  a  passive  thermal  imaging  system  that  produces  an  image  based 
on  temperature  variation  by  detecting  mid-infrared  and  far-infrared  radiation.  Both 
sensors  are  bore  sighted  and  are  packaged  in  a  gimballed  “ball”  that  is  mounted  on  the 
exterior  of  an  aircraft  or  vehicle.  The  use  of  gyros  inside  the  ball  allows  the  camera 
within  to  maintain  its  orientation  in  the  Earth  frame  of  reference,  without  being 
affected  by  roll,  pitch  and  yaw  changes  in  the  supporting  aircraft  or  vehicle. 

In  order  to  ensure  that  a  SAR  operator  would  be  able  to  use  the  system  effectively  and 
with  a  minimal  amount  of  training,  a  prototype  HMI  was  developed  to  evaluate  design 
concepts.  The  VAPS  HMI  prototype  (ELVISS),  developed  for  the  Silicon  Graphics, 
Inc.’s  (SGI)  platform,  provided  a  cost  effective  method  for  evaluating  the  impact  of 
design  characteristics  of  dual  sensor  systems  on  operator  performance.  However  the 
capability  was  limited  and  the  architecture  was  not  designed  to  support  systematic 
investigation  of  the  usefulness  of  the  proposed  system  under  different  conditions  or  to 
manipulate  the  sensor  and  interface  characteristics. 

The  ELVISS  VAPS  prototype  was  therefore  extensively  enhanced  to  allow  the 
empirical  investigation  of  different  interface  and  sensor  characteristics  on  search  and 
detection  capability  under  different  environmental  conditions.  Included  in  this  upgrade 
was  a  Scenario  Generation  Environment  (SGE)  that  provided  user-friendly  capability 
for  generating  scenarios.  Nonetheless,  despite  the  increased  versatility  of  the  prototype 
the  requirements  of  a  specific  experimental  design  required  that  LUA  scripting  be  used 
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to  make  additional  modifications  to  the  software.  While  further  development 
drastically  expanded  the  LUA  scripting  capabilities,  the  SGE  was  not  similarly 
extended  and  some  of  its  scenario-generation  capabilities  became  incompatible  with 
the  prototype.  Thus  LUA  scripting  in  now  the  primary  mode  of  control  of  the 
prototype. 

The  prototype  HMI  was  then  ported  to  run  on  Microsoft  Windows  XP,  and  required 
replacing  the  VAPS  and  SGI  Performer™  with  equivalent  functionality  using  the 
OpenSceneGraph  open  source  graphics  library.  The  robustness  and  traceability  of  the 
system  were  also  significantly  improved.  The  latest  development  phase  added 
important  tracking  and  motion-related  functionality  (amongst  other  things)  to  this  new 
system  and  gave  it  a  new  name  AIMSsim. 


Aim 

The  aim  of  this  manual  is  to  provide  an  overview  of  the  system  software  developed  to 

support  empirical  investigation  of  a  simulated  user  interface  for  the  AIMS  system. 

Objectives 

The  specific  objectives  of  this  manual  are  to: 

a.  Describe  the  capabilities  and  limitations  of  the  software  developed  to  support 
the  conduct  of  experimentation  using  the  AIMS  software  prototype; 

b.  Define  the  minimum  requirements  of  the  hardware  and  software  environment 
needed  to  utilize  the  software  developed  for  this  project;  and 

c.  Provide  information  about  various  subsystems  and  algorithms  used. 

Report  Outline 

The  report  is  structured  as  follows: 

a.  Section  One  describes  the  background,  aim  and  scope  of  this  document; 

b.  Section  Two  describes  the  architecture  of  the  developed  software; 

c.  Section  Three  and  Four  explain  capabilities  and  the  limitations,  respectively, 
of  the  developed  software; 

d.  Section  Five  defines  the  hardware  and  software  requirements  for  the  execution 
and/or  modification  of  the  developed  software; 

e.  Section  Six  documents  the  subsystems  and  some  algorithms  of  the  developed 
software;  and 
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f.  The  annexes  document  abbreviations  used  in  this  manual  and  a  support  tool 
called  the  “Scenario  Generation  Environment”. 
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2  Architecture 


The  AIMSsim  is  divided  into  three  separate  applications,  as  sketched  in  Figure  1.  The 
simControl  application,  as  the  name  would  suggest,  controls  the  execution  of  the 
experiment  and  simulation.  It  provides  a  state  machine  and  a  LUA  scripting  interface 
to  describe  experiments,  and  provides  data  collection  facilities.  Also,  it  “models”  the 
world  physics  relevant  to  the  simulation,  such  as  vehicle  (aircraft  and  targets)  motion, 
and  is  the  first  responder  to  operator  controls. 

The  simDisplay  application  is  an  implementation  of  the  AIMS  User  Interface  (UI), 
built  using  OpenSceneGraph  (which  itself  sits  atop  OpenGL).  It  handles  all  visual 
aspects  of  the  HMI:  rendering  the  sensor  displays  and  the  moving  map  display 
(MMD),  providing  dialog  screens,  etc.  In  the  displays,  it  is  in  charge  of  computing  and 
overlaying  such  information  as  the  current  zoom  factor,  the  simulation  time,  the  height 
above  terrain,  the  map  scale,  the  sensor  footprint  or  search  history,  etc. 

The  third  application  is  simlnputs,  which  is  in  charge  of  reading  the  hardware  via  a 
serial  port  connection,  packages  the  data  and  makes  it  available  to  simControl  via 
shared  memory. 


Figure  1:  AIMSsim  architecture 

The  behavior  of  the  UI  is  controlled  remotely  by  simControl,  and  as  such  simDisplay 
can  be  thought  of  as  a  display-only  application.  It  has  no  knowledge  of  how  the  virtual 
world  behaves,  only  how  it  looks  visually.  However  it  is  more  than  a  “dumb  display” 
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since  it  is  able  to  compute  quantities  related  to  geometry,  such  as  line-of-sight 
intersection  with  the  terrain. 

Both  simlnputs  and  simDisplay  are  automatically  started  when  simControl  is  run. 
Hardware  state  data  from  simlnputs  is  read  by  simControl  via  shared  memory.  The 
state  of  the  world  and  what  type  of  information  or  screen  to  show  in  simDisplay  is  also 
communicated  from  simControl  to  simDisplay  via  shared  memory.  In  some  instances, 
simDisplay  makes  some  of  its  computations  available  to  simControl  via  a  read-only 
segment  of  shared  memory,  for  use  by  the  experiment  scripts  for  data  acquisition  or 
decision  making.  Note  however  that  simControl  uses  its  own  copy  of  the  terrain 
geometry  for  time-critical  computations.  Finally,  simControl  and  simDisplay  send  and 
receive  events  from  each  other  via  POSIX  pipes  (one  in  each  direction).  Events 
include  those  resulting  from  operator  input  via  mouse  or  hardware  (pressing  a  trigger 
etc),  those  generated  by  the  experiment  scripts,  and  system  events  indicating 
something  has  happened. 

The  HMI  prototype  is  controlled  by  an  initialization  script  that  defines  the  flight  path, 
runtime  settings,  targets,  and  the  experiment  via  a  programmable  Finite  State  Machine 
(FSM).  This  script  uses  LUA  as  the  scripting  language.  LUA  is  a  relatively  simple 
language  that  is  interpreted  at  run  time  and  uses  a  C-like  syntax.  While  it  is  easy  to 
use,  it  is  at  the  same  time  a  very  powerful  language.  For  information  about  LUA  look 
at  the  appendix  in  the  existing  AIMS  User’s  Guide,  or  visit  www.lua.org  for  official 
documentation. 

The  FSM  itself  defines  which  scripts  to  run  when  certain  events  occur,  and  what  is  the 
resulting  state  name  of  the  system.  Part  of  a  script  is  shown  in  Figure  2  as  an  example. 


Set("baseTerrain",  "Nerepis/shaded/ELVISS  ne  smaller.ive") 
PathPlanCreate("aircraftPath") 

PathPlanAddWaypoint("aircraftPath",  7946,  8434,  584,  100) 
PathPlanAddWaypoint("aircraftPath",  6446,  8434,  584,  150,  50) 
PathPlanAddWaypoint("aircraftPath",  6446,  9934,  584,  100) 
PathPlanAddWaypoint("aircraftPath",  7946,  9934,  584,  150,  400) 
PathPlanAddWaypoint("aircraftPath",  7946,  8434,  584,  150) 

AircraftAddPath("aircraftPath") 

ReadTargetsFile("TEST/sge_targets.xml") 

TargetChangeAttrib("target  1",  "retroReflective",  true) 
TargetChangeAttrib("target  3",  "colorOverrideFLIR",  "yes") 
TargetChangeAttrib("target  3",  "colorlnFLIR",  0) 


Figure  2.  Example  of  part  of  a  LUA  initialization  script 

The  visual  database  used  by  simDisplay  includes  terrain  geometry,  targets  geometry, 
and  shader  programs.  The  latter  are  text  files  that  are  used  to  model  various  aspects  of 
sensor  displays.  For  instance,  the  effects  of  fog,  laser  illuminator,  noise,  retro- 
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reflectivity,  and  thermal  imaging  are  generated  via  OpenGL  shaders.  It  is  relatively 
straightforward  to  change  those  text  files  to  obtain  new  visual  effects,  without  having 
to  rebuild  the  software. 
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3  Capabilities 


3.1  General 

The  HMI  prototype,  AIMSsim,  allows  the  dynamic  configuration  and  conduct  of  an 
experimental  Search  and  Rescue  scenario  via  LUA  scripts  and  a  Finite  State  Machine 
(FSM),  and  provides  data  collection  capabilities.  An  AIMSsim  Scenario  Generation 
Environment  (SGE)  is  available  to  ease  the  generation  of  path  plans  and  the 
positioning  of  targets. 


3.2  AIMS  Human  Machine  Interface  (HMI)  Prototype 

The  AIMS  HMI  Prototype  simulates  the  AIMS  user  interface.  The  HMI  Prototype 
utilizes  text  scripts  to  construct  and  execute  a  simulated  scenario.  A  simulated 
scenario  can  involve 

•  the  visualization  of  a  scenario  landscape  (  or  terrain)  as  viewed  through  a 
simulated  AGTV  and  simulated  FLIR 

•  the  visualization  of  a  moving  map  display  (MMD) 

•  the  definition  of  path  plans  that  can  be  strung  together 

•  the  population  of  the  terrain  with  static  and  moving  target  objects,  using  those 
path  plans 

•  the  flight  of  a  simulated  search  aircraft  over  the  terrain,  using  any  sequence  of 
the  defined  path  plans 

•  the  configuration  and  layout  of  the  HMI  itself,  e.g.  whether  the  polar  plot  of 
the  Aircraft  should  be  visible,  etc. 

The  MMD  is  able  to  show  such  information  as  the  sensor  footprint,  search  history, 
map  scale  in  use,  air-to-ground  level,  and  designation  markers,  whereas  the  sensor 
displays  simulate  the  effects  of  noise,  fog,  laser  illuminator,  and  are  overlaid  with 
useful  information  such  as  the  current  zoom  factor,  depth  of  line  of  sight  to  terrain,  etc. 

The  HMI  Prototype  incoiporates  a  powerful  scripting  engine  providing  a 
programmable  Finite  State  Machine  (FSM)  that  can  be  defined  at  start-up,  advanced 
data  collection  capabilities,  dynamic  scenarios  that  can  change  as  the  experiment 
progresses,  and  the  potential  for  external  control  via  Semi- Automated  Forces  (SAF) 
components.  The  data  collection  capability  allows  the  experimenter  to  capture 
important  information  about  the  system  and  interaction  between  the  operator  and  the 
prototype  throughout  the  execution  of  an  experimental  scenario,  to  further  process  the 
data  in  any  way  that  can  be  described  by  the  scripting  language,  and  to  save  it  in  any 
format. 
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The  FSM  allows  for  a  structured  decomposition  of  the  experiment  into  a  series  of 
repeatable  and  reusable  steps.  Functionality  from  one  experiment  can  be  re-used  by 
other  experiments,  with  a  minimal  amount  of  work. 

A  FSM  can  be  viewed  as  a  graph  with  the  nodes  representing  states,  and  the  arcs 
representing  state  transitions,  as  shown  in  Figure  3.  By  describing  all  the  transitions, 
the  FSM,  and  therefore  the  stages  of  an  experiment,  can  be  fully  described.  There  are 
only  two  states  required  by  the  system,  “INIT”  and  “EXIT”,  i.e.  the  start  and  end 
points  of  the  FSM.  All  states  in  between  can  be  named  as  desired  by  the  experimenter. 


Figure  3:  Example  FSM 


The  FSM  that  describes  an  experiment  can  be  set  at  startup  time  through  the  LUA 
initialization  script  given  to  simControl.  This  allows  for  any  ordering  of  screens  and 
behaviour  to  be  created.  The  AIMS  system  should  be  able  to  behave  according  to  any 
desired  experimental  flow  using  the  scripting  facilities  provided.  It  should  be  noted, 
however,  that  care  must  be  taken  in  describing  the  FSM  of  an  experiment,  since  there 
is  no  facility  to  ensure  that  the  FSM  described  makes  sense. 


3.3  Scenario  Generation  Environment  (SGE) 

The  scripting  engine  allows  experiments  to  define  (amongst  other  things)  targets  and 
path  plans,  and  to  specify  simulation  configuration  parameters.  The  SGE  is  a  GUI 
application  that  can  be  used  as  a  visual  aid  to  define  targets  and  place  them  in  the 
world  terrain,  and  to  define  flight  plans,  as  it  provides  ID,  2D,  and  3D  views  of  the 
world  and  targets.  Two  of  the  four  files  generated  by  the  SGE  can  be  read  by 
AIMSsim’s  simControl ,  namely  the  targets  definition  file  and  the  flight  plan  file 
(Figure  4).  The  SGE  also  enables  the  creation  of  several  predefined  types  of  flight 
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plans.  The  flight  plans  created  can  be  loaded  into  path  plans  by  the  AIMSsim  scripts, 
and  used  on  any  vehicle  (aircraft  and  targets),  at  any  time. 


The  SGE  GUI  allows  the  experimenter  to  set  various  simulation  configuration 
parameters,  such  as  the  time-of-day,  FLIR  and  AGTV  noise,  etc.  However,  simControl 
cannot  currently  read  the  simulation  configuration  file  generated  by  the  SGE  (note 
how  there  is  no  arrow  going  from  “sim  config”  box  to  simControl),  so  changing  those 
parameters  through  the  SGE  will  be  a  waste  of  time.  The  only  UI  parts  of  the  SGE  that 
are  useful  to  AIMSsim  experiments  are  the  Targets  pane  and  the  Aircraft  pane. 

Due  to  the  loose  coupling  between  the  SGE  and  the  HMI,  more  details  on  the  SGE 
subsystem  are  given  only  in  the  Annex  B. 
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4  Limitations 


4.1  General 

The  AIMS  HMI  Prototype  has  limitations  that  affect  the  overall  fidelity  of  the  AIMS 
simulation.  Limitations  are  inherent  to  any  computer  simulation  of  a  real  world 
system  and  as  such  should  be  taken  into  consideration  when  employing  the  software  as 
part  of  an  empirical  study. 

The  following  sections  provide  an  itemization  of  the  limitations  of  the  AIMS  HMI 
Prototype.  It  should  be  noted  that  some  limitations  of  the  AIMSsim  software  are  the 
result  of  the  hardware  and  software  associated  with  the  computing  platform  (Microsoft 
Windows  XP,  OpenSceneGraph)  on  which  the  HMI  Prototype  is  currently  hosted. 
These  hardware  and  software  elements  will  be  discussed  in  Section  5  -  Operating 
Requirements. 

4.2  Sensor  Simulation 

The  AIMSsim  software  includes  the  simulation  of  two  electro-optical  sensors:  an 
Active  Gated  TV  (AGTV)  and  a  Forward  Looking  Infrared  (FLIR)  imaging  system. 

In  fact,  the  implementation  of  these  sensors  would  be  best  described  as  “emulation” 
rather  than  “simulation”.  This  is  due  to  the  fact  that  the  sensor-like  imagery  being 
presented  in  the  HMI  Prototype  is  not  the  result  of  a  physics  based  approximation  of 
an  AGTV  and  FLIR  system.  Rather  the  imagery  is  a  “mock-up”  of  the  sensor  imagery 
intended  only  to  provide  the  “look  and  feel”  of  these  electro-optical  sensors  for  the 
purpose  of  providing  a  more  complete  HMI.  As  such,  the  HMI  Prototype  should  not 
be  used  to  measure  the  effectiveness  of  an  AGTV  or  FLIR  sensor. 

E.g.  FLIRs  have  the  ability  to  penetrate  fog  to  some  extent.  The  effect  of  having  fog  in 
the  environment,  if  specified  in  the  LUA  scripts,  does  not  affect  the  FLIR  display.  In 
this  sense,  the  HMI  FLIR  has  perfect  penetration  of  fog.  Better  modeling  of  fog  effects 
in  FLIR  displays  may  be  worth  considering  for  the  future. 

Similarly,  image  degradation  due  to  noise  and  various  physical  phenomena  is 
simulated  very  simply  by  adding  a  random  component  to  the  pixels  in  the  AGTV  and 
FLIR  displays.  However,  this  does  not  take  into  account  the  effect  of  lower  resolution 
of  the  real  displays,  pixilation,  etc. 

4.3  Measurement  Units 

All  quantities  in  the  system,  such  as  distances,  speeds,  altitude,  etc.,  are  neither 
accurate  nor  precise  representations  of  reality.  E.g.  the  aircraft  speed  can  be  anything, 
even  a  speed  faster  than  that  of  light.  All  physical  quantities  defined  for  an  experiment 
are  so  defined  for  their  role  in  supporting  the  objectives  of  the  experiment,  with  no 
presumption  of  fidelity  or  physical  realism. 
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4.4  Targets 

The  AIMS  HMI  Prototype  includes  a  number  of  target  models  (vehicles,  people, 
geometric  primitives).  These  target  models  are  limited  insofar  as  they  do  not  include 
any  thermal  information,  forcing  the  FL1R  to  use  the  same  radiation  information  as  the 
AGTV,  namely  the  visual  spectrum.  For  this  reason,  the  best  FLIR  effect  is  obtained 
for  hot  objects  by  representing  their  hot  part  with  saturated  colors  (e.g.  white). 

An  AIMSsim  Experimental  Scenario  is  currently  limited  to  a  maximum  of  one 
hundred  and  fifty  (150)  targets  and  fifty  (50)  target  types. 

Another  limitation  is  that  targets  that  are  clamped  to  the  ground  will  be  clamped  to  the 
highest  Level-of-Detail  (LOD)  of  the  terrain  model  (see  section  4.7.1  for  a  discussion 
of  LOD).  This  may  lead  to  some  visual  artefacts  if  the  graphics  display  is  using  a 
lower-detail  level  of  the  terrain  database,  such  as  the  target  appearing  slightly  above  or 
below  the  ground. 

Another  limitation  is  that  simDisplay  only  gets  updates  from  simControl  at  irregular 
intervals,  because  both  processes  are  each  working  independently  at  their  own  tasks.  If 
the  display  is  running  faster  than  the  controller,  then  it  is  forced  to  “guess”  where  a 
moving  target  is,  using  dead  reckoning.  Once  it  gets  an  update,  it  corrects  the  target’s 
position.  This  could  lead,  for  fast-moving  targets,  to  what  might  otherwise  appear  as 
erratic  motion. 

4.5  Aircraft  Flight  Simulation 

The  AIMS  HMI  Prototype  simulates  the  movement  of  the  sensor  package  through  the 
simulated  landscape  as  if  it  were  attached  to  an  imaginary  search  aircraft.  The  HMI 
Prototype  does  not,  however,  include  a  flight  simulation  model  to  control  this 
movement.  The  HMI  Prototype  utilizes  a  simplistic  path  navigation  algorithm  that 
allows  the  movement  of  the  simulated  sensor  “viewpoint”  along  straight-line  segments 
between  “waypoints”  defined  in  the  AIMSsim  LUA  scripts.  The  movement  of  the 
viewpoint  is  “smoothed”  in  the  vicinity  of  the  waypoints  so  as  not  to  create  a  sudden 
change  in  direction  as  the  path  navigation  algorithm  transitions  from  one  path  segment 
to  the  next. 

In  real  life,  a  search  and  rescue  (SAR)  aircraft  would  endeavour  to  maintain  a 
consistent  relative  altitude  (altitude  above  ground  level  or  AGL)  as  it  flies  along  a 
search  path.  This  is  not  practical  with  the  simplistic  path  navigation  algorithm  utilized 
by  the  HMI  Prototype.  As  such,  the  HMI  Prototype  is  limited  insofar  as  it  relies  upon 
the  absolute  altitude  (altitude  above  sea  level  or  ASL)  defined  by  the  various 
waypoints  to  control  the  altitude  of  the  simulated  search  aircraft. 

4.6  Scripting 

The  scripting  engine  currently  requires  that  scripts  that  load  other  scripts  in  the  same 
folder  explicitly  specify  the  path  to  those  scripts,  relative  to  the  folder  from  which 
simControl  was  executed.  This  is  a  common  cause  of  bugs  in  the  scripts  when  they  are 
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copied  from  an  existing  experiment  into  a  new  experiment  folder:  any  changes  to  the 
(non-initialization)  scripts  in  the  new  folder  will  not  be  seen  if  the  experiment  forgot  to 
adjust  the  file  paths  in  the  initialization  script,  after  copying  the  experiment. 

Setting  attributes  is  currently  rather  tedious  due  to  the  low-level  interfacing  method 
used  for  the  scripting  engine.  The  scripting  engine  would  benefit  greatly  from  the  use 
of  one  of  the  many  open-source  wrappers  that  automate  the  exposition  of  system 
variables,  such  as  SWIG  or  ToLua++.  E.g.,  instead  of 

Set ( "mapSensorHistoryState" ,  "ENABLED") 

ChangeTargetAttrib ( "target  1",  "isTarget",  "true") 

the  experimenter  could  simply  write 

map . sensorHistoryState  =  ENABLED 
targets [ 1 ]. is_target  =  true 


4.7  Prototype  Performance 

AIMSsim  has  been  optimized  to  maximize  the  runtime  performance  of  the  prototype 
software.  Despite  these  optimizations,  AIMSsim  suffers  from  performance  limitations 
resulting  from  the  graphics  capabilities  of  graphics  cards  available  on  Microsoft 
Windows  platforms.  The  most  noticeable  element  of  this  limited  graphics  capability  is 
the  update  rate  of  the  prototype  and  sensor  images.  AIMSsim  is  currently  capable  of 
achieving  a  maximum  update  rate  of  20  Hz  -  this  falls  short  of  a  desired  update  rate  of 
60  Hz1. 

4.7.1  Effect  of  Terrain 

The  maximum  achievable  update  rate  is  directly  affected  by  the  complexity  of  the 
visual  terrain  database  that  is  employed  for  a  particular  scenario.  The  use  of  simplistic 
terrain  databases  will  result  in  increased  runtime  performance,  while  the  use  of 
complex  terrain  databases  will  result  in  decreased  runtime  performance. 

The  complexity  of  a  terrain  database  merits  further  discussion.  A  terrain  database  (as 
well  as  any  3D  model)  has  two  main  characteristics  that  drive  the  hardware 
requirements  for  the  computing  platform  on  which  it  will  be  rendered:  the  number  of 
polygons  used  to  construct  the  model,  and  the  amount  of  texture  utilized  by  the  model. 
Both  of  these  should  therefore  as  small  as  possible,  such  that  the  experiment  remains 
realistic. 

Both  the  number  of  polygons  and  the  amount  of  texture  should  be  tailored  to  how 
close  they  are  to  the  viewpoint  (i.e.  the  sensor  on  the  aircraft).  OpenSceneGraph 
supports  the  concept  of  Level-of-Detail  (LOD),  which  allows  several  versions  of  a 
terrain  to  be  in  the  same  terrain  file.  Each  version  has  a  different  degree  of  detail,  such 


1  An  update  of  60  Hz  would  accurately  represent  the  real  life  refresh  rate  of  an  AGTV  and/or  FLIR 
camera.  It  should  be  noted,  however,  that  a  60  Hz  update  rate  was  not  a  requirement  for  the  HMI 
Prototype. 
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that  OpenSceneGraph  will  select  which  version  of  the  terrain  to  show  based  on  the 
distance  between  the  terrain  and  the  viewpoint.  Adding  LOD’s  to  a  terrain  model  can 
improve  performance  dramatically.  However,  as  described  in  an  earlier  section  of  this 
chapter,  it  can  lead  to  some  odd  visual  effects,  such  as  a  target  clamped  to  the  ground 
appearing  to  be  above  the  ground. 


4.7.2  Number  of  Target  Objects 

Similar  to  the  performance  limitations  imposed  by  the  complexity  of  the  terrain 
database,  the  runtime  performance  of  AIMSsim  is  affected  by  the  number  of  target 
objects  present  in  a  scenario.  Since  each  target  object  is  composed  of  a  limited 
number  of  polygons  and  each  target  object  utilizes  a  portion  of  the  system  texture 
memory,  the  rendering  demands  on  the  workstation  are  increased  with  the  addition  of 
target  objects. 

Steps  have  been  taken,  however,  to  mitigate  the  performance  degradation  due  to  an 
increase  in  target  objects.  These  steps  include  the  use  of  Level-of-Detail  (LOD) 
composition  for  target  objects.  LOD  can  be  described  as  a  strategy  for  reducing  the 
number  of  polygons  used  to  represent  a  geometric  object  by  “switching”  between 
numerous  versions  (of  varying  complexity)  of  an  object  at  runtime,  based  on  the 
distance  of  the  viewpoint  from  the  object.  When  the  viewpoint  is  far  away  from  an 
object,  a  representation  of  the  object  with  a  low  number  of  polygons  is  utilized.  As  the 
viewpoint  approaches  the  object,  the  low-polygon  representation  is  replaced  with  a 
more  complex  version  of  the  object  (made  up  of  more  polygons).  By  trading  off  the 
complexity  of  the  representation  with  the  distance  from  the  object,  an  attempt  is  made 
to  balance  the  rendering  capabilities  of  a  system  while  presenting  only  the  information 
that  is  required  to  accurately  represent  the  appearance  of  a  target  object. 

While  this  approach  is  helpful,  it  cannot  address  all  possible  scenarios.  If  for  instance 
a  large  number  of  target  objects  (specifically  those  of  a  complex  nature,  i.e.  a  tank 
model)  were  assembled  in  a  confined  geographic  area,  LOD  would  not  be  helpful 
when  the  user  “zoomed  in”  on  this  area.  This  is  because  the  act  of  “zooming  in”  has 
moved  the  rendering  viewpoint  close  to  the  collection  of  objects  and  forced  them  all  to 
the  highest  level  of  detail. 


4.7.3  Effect  of  Shaders 

Shaders  are  machine  code  created  at  startup  by  the  OpenGL  library  by  compiling 
source  code  stored  in  shader  text  files  into  machine  instruction.  OpenGL  is  the  low- 
level  graphics  API  upon  which  OpenSceneGraph  is  built.  The  use  of  shaders  is  not 
absolutely  necessary  but  adds  tremendous  freedom  in  modeling  the  various  sensors. 
Without  using  shaders,  the  HMI  display  can  be  updated  at  over  60  Hz.  When  using  the 
AGTV  shader,  this  goes  down  to  20  Hz  for  an  800x600  display.  The  combined 
performance  hit  of  having  several  shaders  is  not  linear.  However,  shaders  are  a 
relatively  new  addition  to  OpenGL  (ca.  2004),  so  graphics  card  vendors  are  still 
catching  up  to  the  various  types  of  optimizations  possible  within  the  compiled  shader 
code.  The  performance  of  shaders  should  improve  dramatically  over  the  next  year,  as 
graphics  cards  provide  more  hardware  support  for  OpenGL  Shading  Language. 
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Every  geometry  object  in  the  scene  has  a  default  GLSL  Uniform  named 
“nodeHasTexture”  set  to  true.  Objects  that  do  not  have  texture  will  not  be  visible  in  the 
AGTV  display,  unless  their  geometry  defines  this  GLSL  flag  and  sets  it  to  false.  Not 
all  geometry  modelling  programs  will  allow  you  to  define  a  GLSL  Uniform  as  part  of 
the  geometry. 
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5  Operating  Requirements 


5.1  General 

The  following  table  lists  the  minimum  system  requirements  to  install  and  run  the 
AIMSsim  SGE  and  AIMSsim  HMI  Prototype. 


Table  1:  Minimum  Configuration  Requirements 


HARDWARE 

SOFTWARE 

MEMORY 

CPU:  DUAL  Intel  XEON 

Microsoft  Windows  XP  Pro, 

■  4GB  memory 

3.6GHz,  1MB  Cache,  4GB 

service  pack  2 

RAM,  PCI-Express, 

■  1GB  disk  space  for 

■ 

OpenSceneGraph  0.9.9  or 

geometry,  binaries, 

Graphics:  BFG  6800Ultra 

later,  Expat  1.95.8,  boost 

and  source  code 

Overclock  Geforce  6800 

1.32,  LUA  5.0 

256MB  PCI-Express  Dual  DVI 

■ 

FLTK  1.1.7  (for  the  SGE) 

Monitor:  ViewSonic  VP201b 

Black  20. 1 "  1 600x1 200  HDTV 

Visual  Studio  .Net  2003  (for 

LCD 

modifications  only) 

■  BG  Systems  FlyPanel 


5.2  File  and  Folder  Locations 

Only  the  simControl,  simDisplay  and  SGE  have  certain  requirements: 

•  simControl :  the  simlnputs  and  simDisplay  executables,  and  all  the  DLLs,  must 
be  in  the  system  PATH,  or,  if  not,  must  be  in  the  same  folder  as  simControl. 

•  simDisplay.  the  visuals  database  is  assumed  to  be  located  in  the  folder  named 
in  the  AIMS  DATA  environment  variable,  and  is  assumed  to  have  the 
following  structure: 

AIMS_DATA_FOLDER 
target  map.txt 
shaders 
terrains 
targets 
textures 

All  shaders  are  looked  for  in  the  shaders  folder,  all  terrain  geometry  in  the 
terrains  folder,  all  target  files  specified  in  the  target_map.txt  file  in  the  targets 
folder,  and  all  textures  in  the  textures  folder. 
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•  SGE:  the  scenarios  can  be  stored  anywhere,  however  the  default  location  is  as 
specified  by  AIMS_SCEN. 


5.3  Environment  Variables 

Some  run-time  settings  used  by  AIMSsim  are  independent  of  scenario  and  relate  to 
“application  configuration”  rather  than  “scenario  description”.  Those  setting  have  been 
exposed  through  environment  variables  that  should  be  set  once  in  every  user’s 
environment,  and  need  not  be  changed  again.  The  following  table  summarizes  the 
currently  available  environment  variables  for  the  AIMS  HMI  Prototype: 

Table  2:  Environment  Variables 

Environment  Description  Value 

Variable 

AIMS_DATA  Path  to  the  AIMSsim  database  folder.  This  is  a  full  String 

path  to  the  database  described  in  section  5.2  -  File 
and  Folder  Locations. 

AIMS_SCEN  Path  to  the  AIMSsim  scenarios  folder,  used  by  the  String 

SGE  only. 

AIMS_DPI  Space-separated  pair  of  integers  specifying  dots-  int  int 

per-inch  along  x  and  y  axes  of  the  computer  screen. 

Necessary  for  accurate  rendering  of  moving  map 
display. 

FB_EMULATOR  If  set  to  something  (anything),  the  simlnputs  will  String 

replace  itself  with  a  Java-based  hardware  emulator, 
useful  when  the  FlyPanel  is  not  available.  Note  that 
the  emulator  is  only  available  to  the  staff  at  Greenley 
&  Associates:  it  is  an  aid  to  development  and  is  not 
“production  quality”,  so  it  is  not  supported  externally. 
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6  Algorithms  and  Design 


6.1  Components  and  Dependencies 

Figure  5  represents  the  software  components  of  AIMSsim  and  their  dependencies.  The 
green  boxes  represent  programs  that  can  be  run,  all  others  are  libraries.  The  light  blue 
boxes  are  in-house  libraries,  the  dark  blue  boxes  are  Open  Source  libraries,  typically 
based  on  the  GPL  license. 


Figure  5:  AIMSsim  components  and  their  dependencies 


6.2  Simulation  Loop 

The  simulation  loop  of  simControl  involves  the  following  steps,  which  may  be  useful 
to  know  when  defining  timed  events,  periodic  scripts,  and  other  script  tasks: 

1 .  Get  new  events  from  simDisplay,  put  in  event  queue 

2.  Queue  new  events  for  run-timed  events  and  flight-timed  events  (defined  in 
scripts) 

3.  Copy  shared  memory  written  by  simDisplay  to  local 

4.  Get  new  events  from  simlnputs  (digital  inputs),  put  in  event  queue 

5.  Handle  operator  analog  input 

6.  Update  world  entities  (aircraft  position  and  speed,  targets,  etc) 

7.  Execute  periodic  scripts 
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8.  For  each  event  in  event  queue:  react  to  event  (state  transition,  execute,  etc.) 

9.  Commit  the  new  state  of  the  world  to  shared  memory  read  by  simDisplay 

10.  Send  any  messages  queued  to  simDisplay,  via  pipe 

Similarly,  simDisplay  uses  a  graphics  update  loop  that  does  the  following: 

1.  Copy  shared  memory  written  by  simControI  to  local  (new  state  of  world) 

2.  React  to  any  messages  sent  by  simControI  via  pipe 

3.  Update  the  display 

4.  Commit  any  new  computations  to  shared  memory,  in  case  simControI  needs  the 
results 

5.  Send  any  new  events  queued  by  simDisplay,  to  simControI  via  pipes 

Step  #3,  “update  the  display”,  will  involve  more  or  less  work  depending  on  what  is 
visible  in  the  GUI:  a  dialog  screen  (e.g.  the  intro,  start  or  exit  screens),  or  the 
operational  view.  It  basically  consists  of: 

1 .  update  the  viewport  dimensions  (in  case  user  has  resized  window) 

2.  handle  keyboard  and  mouse  events 

3.  for  each  subdisplay  in  display: 

a.  update  frame  stamp 

b.  update  the  scene  graph 

c.  cull 

d.  draw 

Note  that  a  dialog  panel  only  has  one  subdisplay,  whereas  the  operational  view  has 
five  (5). 

6.3  Inter-process  synchronization 

SimControI  and  simDisplay  are  independent  processes  that  communicate  via  pipes  and 
shared  memory.  Each  process  uses  atomic  read/write  operations  on  the  whole  shared 
memory  segment  at  once,  at  the  beginning  or  end  of  each  simulation/graphics  loop,  so 
as  to  guarantee  that  the  shared  state  is  self  consistent  and  doesn’t  contain  a  mixture  of 
old  and  new  data. 

The  separation  of  data  exchange  between  the  two  processes  between  pipes  and  shared 
memory  is  somewhat  unfortunate,  since  there  is  no  synchronicity  between  pipes  and 
shared  memory.  E.g.,  if  simDisplay  writes  a  message  to  a  pipe,  and  immediately  after 
it  commits  its  new  computations  to  shared  memory,  simControI  may  still  see  the 
shared  memory  update  before  it  sees  the  pipe  message.  This  has  not  been  observed  in 
practice  but  it  is  a  possibility.  The  consequence  is  important  only  when  the  message 
being  sent  is  related  to  the  state  being  updated,  as  happens  during  an  “isect  request”  to 
the  simDisplay:  simDisplay  computes  the  isect,  puts  it  in  shared  memory,  and  sends  an 
event  that  the  isection  result  is  available.  SimControI  may  get  the  “isection  valid” 
event  before  its  shared  memory  has  been  updated,  which  could  cause  the  script  that 
gets  run  when  this  event  takes  place  to  use  garbage  isection  data. 
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6.4  Noise/Degradation 

The  degradation  effect  in  the  sensor  displays  uses  a  texture  created  at  runtime  from  a 
fixed  random  seed.  The  texture  coordinate  of  the  active  texture  is  used  to  look  up  the 
noise  texture  color,  for  every  image  fragment.  The  noise  is  greyscale  and  stored  in  the 
shader  file  noise2DRed.frag.  It  requires  only  the  noise  amplitude  and  a  random  offset 
so  that  the  noise  appears  to  change  in  time  on  the  display.  See  the  applyNoiseQ 
functions  defined  in  the  shader  file  for  more  details.  A  better  technique  would  be  to 
use  in-memory  rendering  of  the  scenery  image,  and  apply  a  noise  function  to  the 
image,  before  displaying  it. 

6.5  Fog 

The  prototype  can  provide  a  distance  based  fog  to  restrict  the  viewing  distance  of  the 
sensor  images.  This  fog  is  a  per-pixel  linear  OpenGL  Fog.  However  it  is  implemented 
in  the  AGTV  shader,  hence  it  could  be  improved  easily  without  requiring  any  software 
modifications.  The  parameters  for  the  fog  can  be  set  using  the  fogDistance, fogOnset, 
and  fogColor  LUA  variables.  The  first  variable  sets  the  maximum  viewable  distance, 
in  metres,  and  the  second  sets  the  near  edge  of  fog.  If fogDistance  is  not  set,  no  fog  is 
provided.  The  fog  color,  greyscale,  is  the  product  of  the  timeOfDay  value  and 
fogColor  value. 

6.6  LOD(TFLOD) 

The  simulation  provides  a  few  methods  to  tune  level  of  detail  used  for  the  terrain. 

First,  the  agtvLODScale  and  flirLODScale  LUA  variables  allow  the  LOD  scale  for  the 
simulations  to  be  set  individually.  A  value  of  1  means  the  LOD  will  behave  as 
specified  by  the  terrain.  A  value  less  than  1  will  increase  the  switching  distance,  a 
value  of  more  than  1  will  decrease  the  switching  distance.  A  value  of  0  means  that  the 
highest  level  of  detail  will  always  be  used.  See  the  OpenGL  Performer  documentation 
for  a  more  complete  description  of  this  feature. 

On  computers  with  a  graphics  card  that  supports  Terrain  Fade  Level  of  Detail 
(TFLOD),  this  option  will  be  enabled  by  default.  This  option  smoothes  the  transition 
between  levels  of  detail. 

6.7  AGTV  Illuminator 

The  AGTV  illuminator  is  modelled  differently  if  fog  is  present.  Without  fog,  the 
illuminator  is  assumed  to  be  on  with  auto-gain  control  such  that  the  area  of  the  display 
outside  the  illuminated  area  appears  dimmed,  while  the  scene  in  the  illuminated  area  is 
untouched. 

When  fog  is  toggled  on,  the  illuminator  is  modelled  as  “seeing  through”  the  fog.  The 
illuminated  area  appears  without  fog,  the  non-illuminated  area  appears  with  normal, 
undimmed  fog. 
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Objects  that  are  retro-reflective  will  become  saturated  white  when  they  enter  the 
illuminated  area. 


6.8  Auto-tracking 

The  auto-tracking  feature  is  implemented  via  LUA  scripting.  It  currently  makes  use  of 
AIMSsim  exported  functions  to  reset  periodically  (several  times  per  second)  the 
camera  pod  heading  and  pitch,  so  as  to  be  “pointing”  towards  the  ground  position  that 
was  in  the  Line  of  Sight  at  the  time  auto-tracking  was  started.  It  is  therefore  only  an 
approximation: 

•  The  center  point  of  the  AGTV  display  must  be  on  some  object  (the  sky  won’t 
work) 

•  If  the  target  that  is  being  auto-tracked  becomes  hidden  behind  a  mountain  the 
tracking  will  not  be  lost 

•  The  “tracker”  works  independently  from  the  AGTV  or  FLIR  displays  and  does 
not  depend  on  noise  level  etc 

As  well,  setting  the  heading  and  pitch  of  the  camera  pod  can  cause  a  small  amount  of 
jitter,  since  small  numerical  errors  in  the  angles  will  result  in  large  variations  of  the 
line  of  sight  intersection  on  the  terrain.  AIMSsim  provides  a  mode  to  directly  set  the 
“point  to  look  at”  instead  of  setting  the  heading  and  pitch. 

6.9  Terrain  clamping  and  following  for  targets 

All  targets  get  automatically  clamped  to  terrain  underneath  or  above  them  when  added 
to  the  world  via  the  LUA  initialization  script.  Clamping  refers  to  changing  their  Z 
coordinate  so  as  to  position  them  on  the  ground.  This  allows  your  scenario  definition 
to  not  worry  about  target  height  and  makes  it  adaptable  to  different  terrain  databases. 
Targets  that  are  moved  in  the  XY  plane  (via  scripting)  are  automatically  re-clampled 
to  the  ground  at  their  new  lat-lon  position.  This  means  that  targets  can  never  be 
floating  in  mid-air. 

In  the  most  recent  developments,  simControl  was  given  the  ability  to  query  a  geometry 
database  to  find  out  where  the  targets  should  be  clamped.  This  allows  scripts  to  know 
immediately  (without  having  to  wait  for  simDisplay  to  do  the  computation)  where  a 
target  is  going  to  be  clamped. 

Note  that  the  current  implementation  uses  OpenSceneGraph  to  implement  the  database 
reader,  and  do  the  computations  of  terrain  height  at  a  given  position.  Technically, 
simControl  is  completely  unaware  of  this  fact:  from  its  point  of  view,  it  is  using  a 
library  that  has  the  ability  to  read  a  geometry  file  and  return  height  at  a  given  XY.  This 
library,  called  geolsection,  uses  the  highest  level  of  detail  available  from  the  terrain 
database,  because  this  is  how  OpenSceneGraph  operates.  SimControl  doesn’t  care  and 
doesn’t  know. 
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SimDisplay  currently  uses  the  same  library,  geolsection,  to  clamp  the  targets  to  the 
ground,  independently  of  simControl.  It  does  this  because  it  uses  dead  reckoning  to 
estimate  where  the  target  probably  is  in  the  XY  plane,  if  an  update  from  simControl  is 
not  yet  available.  Unfortunately,  OpenSceneGraph  uses  the  highest  level  of  detail 
available  from  the  terrain  database  when  doing  the  geometry  calculations.  This  may  be 
a  different  level  of  detail  than  seen  in  the  operational  screen  by  the  human  operator, 
since  the  level  of  detail  in  use  visually  is  controlled  by  the  underlying  graphics  library 
scene  viewer,  based  on  the  distance  of  the  viewpoint  from  the  observed  geometry. 
There  is  hence  the  potential  for  the  target  to  disappear  under  the  ground  at  the  lower 
level  of  detail.  This  will  not,  in  general,  be  a  problem,  but  should  be  kept  in  mind  if 
some  odd  behaviour  is  observed. 

Another  consequence  of  separation  of  model  ( simControl )  and  view  ( simDisplay )  is 
that  operations  based  on  “designating”  a  target  (such  operations  take  place  in  the 
scripts  in  simControl)  could  fail  if  the  target  is  visible  at  a  different  height  due  to  the 
dead  reckoning,  level  of  detail  used  in  the  visuals,  etc.  This  is  an  inherent  limitation  of 
any  simulation  system  with  this  kind  of  architecture. 

Finally,  note  that  targets  moved  along  a  non-flat  ground  cannot  maintain  a  constant 
speed,  because  the  speed  affects  how  “far”  along  the  surface  the  target  can  move,  yet 
the  slope  of  the  surface  changes  at  every  point  of  the  “journey”. 
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Annex  A:  List  of  abbreviations  and  acronyms 


AGTV 

AIMS 

AIMSsim 

ASCII 

Active  Gated  TV 

Advanced  Integrated  Multi-sensor  Surveillance 

AIMS  simulator 

American  Standard  Code  for  Information  Interchange 

DND 

DRDC 

Department  of  National  Defence 

Defence  Research  and  Development  Canada 

ELVISS 

Enhanced  Fow-Fight  Fevel  Visible  and  Infrared  Surveillance  System 

FLIR 

FLTK 

FOV 

Forward  Fooking  Infrared 

Fast  Fight  Tool  Kit 

Field  of  View 

HMI 

Human  Machine  Interface 

IR 

Infrared 

MMD 

Moving  Map  Display 

N/A 

Not  Applicable 

PC 

Personal  Computer 

SAR 

SGE 

SGI 

Search  and  Rescue 

Scenario  Generation  Environment 

Silicon  Graphics,  Inc. 

VAPS 

VPI 

Virtual  Applications  Prototyping  System 

Virtual  Prototypes,  Inc. 

Wx 

Weather 
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Annex  B:  Scenario  Generation  Environment  (SGE) 


The  SGE  is  an  ideal  environment  in  which  to  create  flight  paths  and  position  targets  as 
it  gives  a  visual  representation  of  a  specified  terrain,  targets,  and  a  flight  plan,  and  has 
several  types  of  flight  plans  predefined  and  parameterized. 

The  current  version  of  the  SGE  creates  4  data  files  in  a  XML  format.  Only  two  of 
these  files  can  be  read  directly  by  the  AIMSsim  application:  the  flight  plan  file,  whose 
parameters  are  edited  on  the  “Aircraft”  panel,  and  the  targets  definitions  file,  whose 
parameters  are  edited  on  the  “Targets”  panel.  Both  panels  are  described  in  the 
following  sections,  B.l  and  B.2.  All  other  panels  contain  parameters  that  cannot  be 
read  by  AIMSsim,  so  those  panels  are  of  little  use  for  experiment  development.  They 
are  described  briefly  in  this  Annex’s  sections  B.3  and  B.4  only  for  historical  reasons, 
and  should  not  be  used  as  a  reference. 

The  SGE  comprises  a  graphical  user  interface  (GUI)  featuring  a  moving-map  display 
of  the  geographic  scenario  area,  as  well  as  graphical  controls  providing  the  means  to 
manipulate  scenario  parameters.  The  SGE  GUI  is  depicted  in  Figure  6. 


Figure  6.  SGE  User  Interface 
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The  SGE  allows  the  experimenter  to  control  the  following  scenario  elements: 


a.  Position  and  appearance  of  target  objects  on  the  scenario  landscape; 

b.  Flight  plan  of  the  simulated  search  aircraft; 

c.  Configuration  of  the  sensor  package  and  HMI  screen  layout;  and 

d.  Environmental  parameters  controlling  the  effectiveness  of  the  simulated 
sensors. 

In  addition,  the  SGE  provides  the  following  features: 

a.  Real-time  manipulation  of  the  moving- map  display  (zoom  and  pan); 

b.  Graphical  depiction  of  the  aircraft  altitude  profile  throughout  the  scenario; 

c.  Estimation  of  scenario  duration;  and 

d.  Total  number  of  targets  and  non-targets  defined  in  the  scenario. 

B.1  Manipulation  of  Targets 

The  SGE  provides  an  interface  whereby  an  experimenter  may  manipulate  target 
objects  that  will  form  part  of  an  experimental  scenario.  At  the  highest  level,  the  SGE 
allows  a  user  to  “Add”  and  “Delete”  targets  from  the  scenario.  Once  a  target  has  been 
added  to  the  scenario,  a  user  may  modify  various  parameters  that  affect  the  position 
and  appearance  of  the  target  object  when  viewed  in  the  HMI  prototype.  Table  3 
describes  the  modifiable  target  parameters  and  defines  the  valid  range  of  values  for 
each  parameter  as  well  as  the  units  of  each  parameter,  where  applicable. 


Table  3.  SGE  Target  Parameters 

PARAMETER 

VALID  RANGE 

UNITS 

Name 

Any  alpha-numeric  string 
(maximum  of  TBD  characters) 

N/A 

Target 

Boolean  value 

N/A 

Retro_reflective 

Boolean  Value 

N/A 

Type 

Pick  list  (see  Appendix  A  of  the 

User  Manual  for  a  list) 

N/A 

Visibility 

Pick  list 

(Both,  AGTV  Only,  FLIR  Only) 

N/A 

Orientation 

0  to  359 

Degrees 

Identifier 

Boolean  value 

N/A 

Identifier  Label 

Pick  list  (A  to  F,  0  to  9) 

N/A 
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Position  (x,  y,  z) 

Single  precision  floating  point 
value 

Meters 

AGTV  Colour  Adjust 

Boolean  value 

N/A 

AGTV  Colour 

Single  precision  floating  point 
value 

N/A 

FUR  Colour  Adjust 

Boolean  value 

N/A 

FUR  Colour 

Single  precision  floating  point 
value 

N/A 

A  user  may  modify  the  position  of  a  selected  target  by  two  methods:  entering  the 
discrete  values  for  “x”,  “y”  and  “z”  or  by  selecting  a  location  on  the  moving-map 
display,  after  clicking  the  “Select”  button.  When  entering  discrete  values,  the  user 
must  press  ‘Enter’  (after  the  entry  of  each  value)  for  the  parameter  changes  to  be 
registered  to  the  target  in  question. 


B.2  Manipulation  of  Flight  Plan 

The  SGE  allows  the  experimenter  to  modify  the  sequence  of  waypoints  (or  flight 
plan)  of  a  simulated  search  aircraft  to  which  the  simulated  sensor  package  is  mounted. 
The  flight  plan  interface  allows  the  experimenter  to  select  between  three  different 
methods  of  creating  a  flight  plan.  They  are: 

a.  User  defined  flight  plan.  The  user  may  interactively  “Add”  and  “Delete” 
waypoints  from  the  flight  plan.  For  each  waypoint  the  user  may  modify  the 
position  (x,  y,  altitude)  and  the  speed  of  the  aircraft  at  that  waypoint. 

b.  Creeping  line  search  pattern.  The  user  may  create  a  flight  plan  based  on  an 
algorithm  that  builds  a  “Creeping  Line  Ahead”  flight  pattern.  The  user 
selects  a  “Start  Point”  for  the  pattern  and  provides  additional  parameters 
necessary  for  the  SGE  to  automatically  construct  the  flight  plan. 

c.  Expanding  square  search  pattern.  The  user  may  create  a  flight  plan  based  on 
an  algorithm  that  builds  an  “Expanding  Square”  flight  pattern.  The  user 
selects  a  “Start  Point”  for  the  pattern  and  provides  additional  parameters 
necessary  for  the  SGE  to  automatically  construct  the  flight  plan 

Tables  4,  5  and  Table  6  describe  the  modifiable  flight  plan  parameters  for  each  of  the 
flight  plan  creation  methods  mentioned  above. 
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Table  4.  SGE  User  Defined  Flight  Plan  Parameters 


PARAMETER 

VALID  RANGE 

UNITS 

Waypoint  Position 
(x,  y,  altitude) 

Single  precision  floating  point 
value 

Meters 

Speed 

0  to  200 

Knots 

‘These  parameters  apply  to  each  waypoint  in  the  flight  plan. 

Table  5.  SGE  Creeping  Line  Flight  Plan  Parameters 

PARAMETER 

VALID  RANGE 

UNITS 

Start  Position 
(x,  y,  altitude) 

Single  precision  floating  point 
value 

Meters 

Speed 

0  to  200 

Knots 

S 

0.1  to  10.0 

Kilometres 

Leg  Length 

0.1  to  20.0 

Kilometres 

#  of  Legs 

1  to  150 

N/A 

Table  6.  SGE  Expanding  Square  Flight  Plan  Parameters 

PARAMETER 

VALID  RANGE 

UNITS 

Start  Position 
(x,  y,  altitude) 

Single  precision  floating  point 
value 

Meters 

Speed 

0  to  200 

Knots 

S 

0.1  to  10.0 

Kilometres 

#  of  Legs 

1  to  150 

N/A 

Direction 

Pick  list  (CW,  CCW) 

N/A 

B.3  Manipulation  of  Sensors 

The  SGE  provides  the  user  with  the  capability  to  modify  the  characteristics  of  the 
simulated  AIMS  sensors.  These  characteristics  affect  the  window  layout 
configuration  to  be  used  by  the  HMI  prototype,  as  well  as  the  Field  Of  View  (FOV) 
or  zoom  of  the  individual  sensors.  Table  7  describes  the  modifiable  sensor 
parameters  and  defines  the  valid  range  values  and  units  for  each  parameter,  as 
applicable. 
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Table  7.  SGE  Sensor  Parameters 


PARAMETER 

VALID  RANGE 

UNITS 

Sensor  Window 
Configuration 

Pick  list  (Both  Equal,  AGTV 
Primary,  FUR  Primary,  AGTV 

Only,  FUR  Only) 

N/A 

Slave  Sensor  FOV 

Boolean  value 

N/A 

AGTV  Zoom  (min/max) 

0.5  to  40.0 

Degrees 

AGTV  Zoom  Method 

(Continuous/  Discrete) 

N/A 

FUR  Zoom  (min/max) 

0.5  to  40.0 

Degrees 

FUR  Zoom  Method 

(Continuous/  Discrete) 

N/A 

Simulate  CCD 

Boolean  value 

N/A 

Default  AGTV  Beam 

Width 

Pick  list  (Wide,  Narrow) 

N/A 

AGTV  Narrow  Beam 

Size 

Pick  list  (2,  5) 

Degrees 

AGTV  Wide  Beam  Size 

Pick  list  (10,  15,  20,25,  30,35) 

Degrees 

B.4  Manipulation  of  Additional  Scenario  Parameters 

The  SGE  allows  you  to  control  additional  scenario  settings.  These  miscellaneous 
settings  control  the  default  configuration  of  the  HMI  Prototype  display  and  of  the 
HMI  control  hardware.  Table  8,  Table  9,  and  Table  10  describe  the  modifiable 
miscellaneous  scenario  parameters  and  define  the  valid  range  values  and  units  for 


each  parameter,  as  applicable. 

Table  8.  SGE  Map  Parameters 

PARAMETER  VALID  RANGE  UNITS 

Map  Mode  Pick  list  (No  Map,  2D  Paper,  2D  N/A 

Terrain,  2D  Shaded,  Immersed, 

Immersed  Shaded,  Tethered, 

Tethered  Shaded) 

MapScale  Range  (1:1000,  1:250000)  N/A 

Map  Orientation  Pick  list  (North  Up,  Aircraft  Up,  N/A 

Camera  Up) 

Aircraft  Symbol  Pick  list  (Rotary  Wing,  Fixed  Wing,  N/A 

Pointer,  No  Icon) 


28 


DRDC  Atlantic  CR  2006-279 


Map  Type 

Pick  List 

N/A 

Colour 

Pick  List(Colour,  Greyscale) 

N/A 

Map  Slave  FOV  to 

Sensor 

Boolean 

N/A 

Map  FOV 

1  to  120 

Degrees 

Table  9.  SGE  Environment  Parameters 

PARAMETER 

VALID  RANGE 

UNITS 

Time  of  Day  (TOD) 

0.00  to  1.00 

N/A 

Degradation  Visibility 

Pick  List  (Clear  Degraded) 

N/A 

Degradation  AGTV 
effect 

0.00  to  1.00 

N/A 

Degradation  FUR  effect 

0.00  to  1.00 

N/A 

Table  10.  SGE  Miscellaneous  Parameters 

PARAMETER 

VALID  RANGE 

UNITS 

Scan  Enabled 

Boolean  value 

N/A 

Scan  Rate 

Pick  list  (1.5,  3.0,  6.0) 

Deg/s 

Scan  Width 

Pick  list  (30,  60,  90) 

Deg 

Joystick  Mode 

Pick  list  (Aircraft,  Cursor) 

N/A 

Zoom  Control  Mode 

Pick  list  (Forward=Zoom  In, 
Forward=Zoom  Out) 

N/A 

Memory  Recall  Mode 

Boolean 

N/A 

B.5  Target  Preview 

The  SGE  provides  the  user  with  the  capability  to  view  the  placement  and  orientation 
of  an  individual  target  on  the  scenario  landscape.  This  is  now  a  mode  of  operation 
from  within  the  main  window  of  the  SGE  application.  This  mode  can  be  toggled  by 
pressing/releasing  the  “view”  button  after  selecting  a  target.  Once  invoked,  the 
selected  target  is  displayed  in  relation  to  the  terrain  where  it  was  placed.  By  using  the 
left  and  right  mouse  buttons,  the  target  (as  well  as  the  corresponding  scene)  can  be 
manipulated  in  3D  space.  In  addition,  the  target’s  orientation  can  be  manipulated 
using  the  orientation  dial  control  in  the  SGE  window.  The  target  view  mode  is 
depicted  in  Figure  7. 
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Figure  7.  SGE  Target  Preview  Application  (Manual  Mode) 


B.6  File  Input/Output 

The  SGE  provides  the  user  with  the  capability  to  “load”  and  “save”  entire  scenarios  as 
well  as  individual  scenario  elements.  This  merits  some  discussion.  A  “scenario”,  as 
defined  in  the  context  of  this  software,  is  comprised  of  several  scenario  elements. 
Scenario  elements  include  a  target  list,  a  flight  plan  and  a  prototype  configuration. 

A  target  list  comprises  a  sequence  of  target  definitions  containing  the  parameters 
defined  in  Table  3.  A  flight  plan  consists  of  a  sequence  of  waypoints  representing  the 
parameters  defined  in  Tables  4,  5  and  Table  6.  A  prototype  configuration  consists  of 
configuration  parameters  defined  in  Table  7,  8,  9  and  10  as  well  as  a  reference  to  a 
terrain  landscape.  By  defining  new  scenario  elements  or  by  mixing  and  matching 
existing  scenario  elements,  the  user  can  create  an  infinite  number  of  scenario 
possibilities. 

The  input/output  file  format  specifications  for  the  various  file  types  supported  by  the 
SGE  are  provided  in  Section  6  of  the  AIMS  Software  Prototype  User  Manual. 
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